Reliability and Security of Deep Neural Networks
¶

No description has been provided for this image No description has been provided for this image No description has been provided for this image
$\color{blue}{\text{Masoud Daneshtalab:}}$ Professor at Mälardalen University and TalTech¶
$\color{blue}{\text{Seyedhamidreza Mousavi:}}$ PhD student at Malardalan University¶
$\color{blue}{\text{Mohammad Hassan Ahmadilivani (Mojtaba):}}$ PhD student at TalTech University¶
No description has been provided for this image No description has been provided for this image No description has been provided for this image
$\color{blue}{\text{Email:}}$ masoud.daneshtalab@mdu.se, seyedhamidreza.mousavi@mdu.se, mohammad.ahmadilivani@taltech.ee¶
No description has been provided for this image No description has been provided for this image

Reliability and security threats to machine learning-based systems¶

image.png

We are going to focus on two threats:

  • Reliability issues (Hardware Faults) RReLU Framework
  • Adversarial input perturbation ProARD Framework

First Part: Reliable ReLU Toolbox (RReLU) To Enhance Resilience of DNNs¶

What is a Deep Neural Network?¶

image.png

The application of DNNs: Object Detection¶

image.png

What is the Soft-errors problem?¶

image.png

image.gif

❓ What is the issue with ReLU?¶

image.png

image.png

Soft-Error and it's effect on the DNNs Accelerators?¶

image.png

image.gif

✅ What is the solution?¶

image.png

❓How can we find the bound value for each ReLU activation function?

❓If the output activation is higher than bound, what should we do?

❓IS there any real situation that changed the activation value a lot?

❓Do we need to find one bound value for each layer or neuron?

Data representation in Neural Network¶

Fixed-Point Number¶

fixed-point.png

Floating-Point Number¶

image.png

Activation Restriction Methods:¶

  • Bounded ReLU activation functions:

image.png

  • Algorithm for finding the bounds:
    • Ranger used the calibration method in a layer-wise way.
    • FT-ClipAct used a heuristic method based on fault-injection in a layer-wise manner.
    • FitAct leveraged an optimisation-based method in a neuron-wise mode.
    • ProAct used a Hybrid activation function (neuron-wise and layer-wise) and progressive training

ProAct provide an Open-Source Framework for all the Methods (RReLU)¶

It includes implementations of the following algorithms:

  1. FitAct: Error Resilient Deep Neural Networks via Fine-Grained Post-Trainable Activation Functions

  2. FT-ClipAct: Resilience Analysis of Deep Neural Networks and Improving their Fault Tolerance using Clipped Activation

  3. Ranger: A Low-cost Fault Corrector for Deep Neural Networks through Range Restriction

  4. ProAct: Progressive Training for Hybrid Clipped Activation Function to Enhance Resilience of DNNs

How do different methods work?¶

Ranger: Find the maximum value for a validation set and use that value as the bound in each ReLU.¶

FT-ClipAct:¶

image.png

FitAct:¶

image.png

ProAct:¶

image.png

Instructions to use RReLU framework¶

Install Packages and Clone the RReLU Github Repository¶

In [ ]:
!pip install fxpmath
!git clone https://github.com/hamidmousavi0/reliable-relu-toolbox.git
%cd reliable-relu-toolbox/

Build Data Loader¶

First, we need to create the Dataset

In [4]:
from rrelu.setup import build_data_loader
data_loader_dict, n_classes = build_data_loader(dataset='cifar10',
                                    batch_size=128, image_size=32)
import matplotlib.pyplot as plt

# Get the first batch from the training data loader
images, labels = next(iter(data_loader_dict['train']))

# Get the class names for CIFAR-10
# This assumes CIFAR-10 is used as specified in the previous cell
class_names = ['airplane', 'automobile', 'bird',
               'cat', 'deer', 'dog', 'frog',
               'horse', 'ship', 'truck']


# plt.figure(figsize=(10,10))
# for i in range(25):
#     plt.subplot(5,5,i+1)
#     plt.xticks([])
#     plt.yticks([])
#     plt.grid(False)
#     # Transpose the image tensor to be in HxWxC format for plotting
#     plt.imshow(images[i].permute(1, 2, 0))
#     # The CIFAR10 labels are indices, so we use the class_names list
#     plt.xlabel(class_names[labels[i].item()])
# plt.show()

Build Model¶

  • Our tool support pre-trained models on CIFAR-10, CIFAR-100, and ImageNet Dataset

  • Cifar-10 and Cifar-100 supported models:

    • resnet20, resnet32, resnet44, resnet56
    • vgg11_bn, vgg13_bn, vgg16_bn, vgg19-bn
    • mobilenetv2_x0_5, mobilenetv2_x0_75
    • shufflenetv2_x1_5
  • ImageNet supported Models:

    • All the models in the PyTorch-hub
In [ ]:
from rrelu.setup import build_model
model = build_model(name='resnet20',dataset='cifar10', 
                    n_classes=n_classes)
print(model)

Evaluate the original Model with ReLU activation function¶

In [ ]:
from metrics import eval_cpu
print(eval_cpu(model, data_loader_dict))

Convert Floating-Point weight values to the Fixed-Point¶

In [ ]:
import torch
from fxpmath import Fxp
with torch.no_grad():
    for name, param in model.named_parameters():
        if param is not None:
            param.copy_(torch.tensor(Fxp(param.clone().cpu().numpy(),
                    True, n_word=32, n_frac=16, n_int=15).get_val(),
                                     dtype=torch.float32,device='cpu'))

Evaluate the Fixed-Point Model¶

In [ ]:
print(eval_cpu(model, data_loader_dict))

Evaluating Reliability of the model¶

In [ ]:
from metrics import eval_fault_cpu
print(eval_fault_cpu(model, data_loader_dict, 
                     1e-6, bitflip='fixed',iterations=5))

Build the model with Reliable ReLU¶

In [ ]:
from rrelu.setup import replace_act
from metrics import eval_fault_cpu
model = replace_act(model, 'zero', 'ranger', data_loader_dict,
                    'layer', 'fixed',False ,'cifar10',is_root=True,)
print(eval_fault_cpu(model, data_loader_dict, 1e-6,
                     bitflip='fixed',iterations=5))

Results of using different methods:¶

image.png

GitHub project: reliable-relu-toolbox

Paper link: ProAct: Progressive Training for Hybrid Clipped Activation Function to Enhance Resilience of DNNs

Second Part: ProARD: Progressive Adversarial Robustness Distillation: Provide Wide Range of Robust Students¶

How about the perturbation in the input data? How can we defend against this type of attacks?¶

image.png

The state-of-the-art method is Adversarial Training.¶

The state-of-the-art methods are:

  • TRADES: Theoretically Principled Trade-off between Robustness and Accuracy
  • ARD: Adversarial Robustness Distillation: Use a Robust Teacher Network to train the student's networks. (What is the issue?)

image.png

Is it possible to train a single robust supernetwork and extract multiple robust subnetworks from it, each tailored to different resource-constrained devices—without requiring retraining?¶

How should we train this super-network?¶

  • Randomly sample subnetworks from the supernetwork, train them independently, and return their updates to the supernetwork using weight sharing.
  • Does it work?

First step: Making Super Dynamic Network:¶

image.png

If we trained the multiple subnetworks inside a supernetwork by randomly subsampling, what would be their accuracy and robustness?¶

image.png

We propose Progressive Adversarial Robustness Distillation (ProARD).

Enabling the efficient one-time training of a dynamic network that supports a diverse range of accurate and robust student networks without requiring retraining.

ProARD: Progressive Adversarial Robustness Distillation: Provide Wide Range of Robust Students¶

image.png

Project code:ProARD: Progressive Adversarial Robustness Distillation: Provide Wide Range of Robust Students

Results¶

image.png

$\color{blue}{\text{Refrences}}$¶

  1. M. H. Ahmadilivani, M. Taheri, J. Raik, M. Daneshtalab, and M. Jenihhin,

“A systematic literature review on hardware reliability assessment methods for deep neural networks,” ACM Computing Surveys, vol. 56, no. 6, pp. 1–39, 2024. 2. L.-H. Hoang, M. A. Hanif, and M. Shafique, “Ft-clipact: Resilience analysis of deep neural networks and improving their fault tolerance using clipped activation,” in 2020 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 2020, pp. 1241–1246.

  1. B. Ghavami, M. Sadati, Z. Fang, and L. Shannon, “Fitact: Error resilient

deep neural networks via fine-grained post-trainable activation functions,” in 2022 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 2022, pp. 1239–1244. 4. Mousavi, Seyedhamidreza, et al. "ProAct: Progressive Training for Hybrid Clipped Activation Function to Enhance Resilience of DNNs." arXiv preprint arXiv:2406.06313 (2024).

  1. Mousavi, Seyedhamidreza, et al. "ProARD: Progressive Adversarial Robustness Distillation: Provide Wide Range of Robust Students " IJCNN (2025).
  2. M. Goldblum, L. Fowl, S. Feizi, and T. Goldstein, “Adversarially

robust distillation,” in Proceedings of the AAAI conference on artificial intelligence, vol. 34, pp. 3996–4003, 2020.

Thank You¶